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As  the  Army  leadership  continues  to  shape  the  Army  Force  Generation 
(ARFORGEN)  concept  to  generate  continuous  readiness  that  supports  strategic 
flexibility  and  depth  to  the  US  Army  warfighters,  the  US  Army  Test  and  Evaluation 
Command  (ATEC)  leadership  continues  to  embrace  and  implement  strategic  changes 
to  facilitate  the  fielding  of  equipment  to  the  warfighters.  ATEC  has  demonstrated 
adaptability  and  flexibility  in  meeting  the  challenges  associated  with  the  traditional 
acquisition  process  and  the  necessities  resulted  from  the  deployment  of  troops  to  Iraq 
and  Afghanistan.  The  fundamental  purpose  of  Test  and  Evaluation  (T&E)  in  the 
Department  of  Defense  (DoD)  and  the  Military  Services  remains  the  same  as 
acquisition  policies  change.  Four  studies  conducted  by  different  groups  between  1999 
and  2006  consistently  revealed  similar  concerns  on  the  nature  of  the  needed  change. 
In  a  22  December  2007  memorandum,  Office  of  the  Secretary  of  Defense  (OSD) 
promulgated  another  round  of  new  T&E  policies.  This  paper  examines  how  ATEC  is 
adjusting  its  culture  to  implement  principles  from  five  of  these  latest  T&E  policies  and 


recommends  some  changes  in  norms  to  solidify  practices  of  these  policies  throughout 
the  community. 


CHANGING  NORMS:  KEEPING  ARMY  TEST  AND  EVALUATION  FLEXIBLE  AND 

ADAPTABLE 

We  will  continue  to  examine  and  challenge  our  most  basic  institutional 
assumptions,  organizational  structure  paradigms,  policies,  and  procedures 
to  better  serve  the  Army 

— Commanding  General, 

Army  Test  and  Evaluation  Command, 
Commander’s  Priorities  for  FY  10-15 

As  the  Army  leadership  continues  to  shape  the  Army  Force  Generation 

(ARFORGEN)  concept  to  generate  continuous  readiness  that  supports  strategic 

flexibility  and  depth  to  the  US  Army  warfighters,  the  US  Army  Test  and  Evaluation 

Command  (ATEC)  leadership  continues  to  embrace  and  implement  strategic  changes 

to  facilitate  the  fielding  of  equipment  to  the  war-fighters.1  As  an  organization,  ATEC  has 

customarily  demonstrated  adaptability  and  flexibility  in  meeting  the  challenges 

associated  with  the  traditional  acquisition  process  and  the  necessities  resulted  from  the 

deployment  of  troops  to  Iraq  and  Afghanistan.  This  paper  highlights  the  organizational 

evolution  of  the  Army  test  and  evaluation  (T&E)  community,  examines  how  ATEC  is 

adjusting  its  culture  to  implement  principles  from  five  of  the  latest  T&E  policies 

throughout  the  Command,  and  recommends  a  path  ahead  for  successful  inclusion  of 

these  principles  in  the  community’s  business  practices. 

Overview 

The  fundamental  purpose  of  Test  and  Evaluation  in  the  Department  of  Defense 
(DoD)  and  the  Military  Services  remains  the  same  as  acquisition  policies  change.  The 
DoD  acquisition  directives  and  instructions  provide  guidelines  that  establish  a  flexible 
management  framework  consistent  with  statutory  requirements.  Four  studies 
conducted  by  different  groups  between  1999  and  2006  consistently  revealed  similar 


concerns  on  the  nature  of  the  needed  change.2  Are  the  policies  inefficient  or  is  it  the 
workforce’s  interpretation  of  the  policies  that  justifies  change?  One  can  argue  that  the 
answer  is  a  combination  of  the  two.  The  lenses  through  which  the  workforce  interpret 
and  implement  these  policies  play  a  significant  role.  Edgar  H.  Schein,  a  founder  of  the 
field  of  organizational  psychology,  summarizes  the  essence  of  culture  as  “the  learned, 
shared,  tacit  assumptions  on  which  people  base  their  daily  behavior.”3  An  expert  on 
business  leadership  and  professor  at  the  Harvard  Business  School,  John  P.  Kotter 
wrote,  “Culture  refers  to  norms  of  behavior  and  shared  values  among  a  group  of 
people.”4  It  is  reasonable  to  say  that  the  cultures  within  the  DoD  organizations  are 
shaped  by  statutory  requirements,  policies,  and  regulations.  However,  the  norms  reflect 
persistent  ways  and  beliefs  adopted  by  employees  and  vary  from  differing  agencies  and 
sectors  of  these  organizations. 

Strategic  communication,  strategic  and  creative  thinking,  as  well  as  consensus 
building  are  essential  norms  to  be  embraced  by  the  T&E  workforce.  Embracing  these 
norms  add  efficiency  to  the  acquisition  community  to  provide  the  needed  equipment  and 
weapon  systems  to  the  service  members.  This  paper  uses  some  of  the  concepts 
prescribed  in  the  eight-stage  change  process  presented  by  John  P.  Kotter  in  his  book 
Leading  Change.5  First,  this  paper  explores  the  historical  roots  of  T&E  in  acquisition  of 
military  systems.  It  presents  a  synopsis  of  the  various  organizational  changes 
implemented  in  the  Army  T&E  domain  beginning  in  1998,  to  include  the  evolution  of 
ATEC.  It  then  summarizes  the  latest  T&E  policies  originated  in  the  requirements  of 
Section  231  of  the  John  Warner  National  Defense  Authorization  Act  for  Fiscal  Year 
2007. 6  This  is  followed  by  a  highlight  of  changes  engineered  in  ATEC’s  processes. 
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Subsequent  paragraphs  analyze  the  scope  of  five  of  the  latest  policies  and  their 
implementation  in  ATEC.  Finally,  the  paper  offers  some  changes  in  norms  for 
consideration  to  solidify  practices  of  these  policies  throughout  the  community.  The 
intent  in  this  paper  is  not  to  advocate  for  ATEC,  as  an  Operational  Test  Agency  (OTA), 
to  relinquish  its  statutory  obligations.  To  the  contrary,  the  recommended  path  ahead  is 
a  series  of  considerations  that  will  lead  to  more  robust  T&E  strategy  and  facilitate  the 
successful  implementation  of  the  latest  policies. 

Historical  Roots  of  T&E  in  Acquisition  of  Military  Systems 

The  genesis  of  the  current  T&E  policies  resides  in  the  Blue  Ribbon  Commission 
on  Defense  Management  report,  and  was  due  to  the  waste  and  lack  of  trust  that 
pervaded  the  acquisition  process  in  the  1980’s.  President  Reagan  established  the 
commission  in  1986  “in  part  because  public  confidence  in  the  effectiveness  of  the 
defense  acquisition  system  has  been  shaken  by  a  spate  of  “horror  stories”-overpriced 
spare  parts,  test  deficiencies,  and  cost  and  schedule  overruns.”7  The  fundamental 
purpose  of  T&E,  which  is  to  inform  the  decision  makers,  in  the  Department  of  Defense 
(DoD)  system  acquisition  process  remains  the  same  as  the  acquisition  policies  change. 
Describing  how  the  successful  commercial  programs  go  about  identifying  and  managing 
risks,  the  Commission  reported,  “Prototyping,  early  operational  testing,  and  red  teaming 
are  used  in  concert  for  the  timely  identification  and  correction  of  problems  unforeseen  at 
a  program’s  start.”8  However,  one  can  argue  that  T&E  policies  have  not  been 
implemented  consistently  throughout  the  T&E  community.  The  professionals  in  the 
operational  T&E  community  who  find  their  mandate  in  the  Title  10  had  always  been 
reluctant  to  accept  data  generated  in  differing  events  and  environments  or  with 
immature  prototypes.  This  reluctance  has  been  based  on  interpretation  of  Title  10  USC 
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Section  139  which  states  “Operational  test  and  evaluation  means  the  field  test,  under 
realistic  combat  conditions,  of  any  item  of  (or  key  component  of)  weapons,  equipment, 
or  munitions  for  use  in  combat  by  typical  military  users;  and  the  evaluation  of  the  results 
of  such  test.”9 

History  of  Change  and  Adaptability  in  ATEC 

Changes  to  its  structure  to  meet  strategic  challenges  are  not  new  to  ATEC.  The 
Army  T&E  community  went  through  a  major  reorganization  starting  with  the  Vice  Chief 
of  Staffs  approval  on  18  November  1998  to  consolidate  developmental  and  operational 
testing.10  This  resulted  in  the  redesignation  of  the  agency  known  as  Operational  Test 
and  Evaluation  Command  (OPTEC)  to  ATEC  on  1  October  1999. 11  This  new  Command 
was  chartered  to  be  the  Army’s  test  and  evaluation  integrator  by  combining  both 
developmental  and  operational  testing  as  well  as  the  evaluation  functions  under  its 
umbrella.12  ATEC  uses  an  integrated  multi-disciplinary  team,  designated  ATEC  System 
Team  (AST),  which  consists  of  representatives  from  each  subordinate  command  activity 
to  perform  the  T&E  functions  for  each  system.13 

The  leadership  throughout  the  command  sensed  the  need  for  deeper  adjustment 
in  their  command’s  business  practices.  MG  John  Marcello,  CG  ATEC,  referring  to  the 
transformation  undertaken  by  the  Army  to  meet  the  challenges  of  21st  century  and  the 
T&E  community’s  role  in  system  procurement,  wrote  “To  do  its  part,  T&E,  “the 
conscience  of  acquisition,”  must  adjust  the  way  it  does  business,  or  risk  denying  the 
Army’s  grand  initiative  its  needed  momentum.”14  Mr.  Brian  M.  Simmons,  then  Deputy 
Commander  and  Technical  Director  at  the  U.S.  Army  DTC  wrote  in  an  editorial  for  the 
International  T&E  Association  magazine  “The  Office  of  the  Secretary  of  Defense  (OSD) 
will  provide  the  necessary  policy  framework,  budget  numbers  and  requisite  support;  but 
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the  onus  is  on  the  leaders  in  the  field  to  transform  now  to  a  more  flexible,  responsive 
and  efficient  T&E  capability  to  support  the  warfighter-today’s  and  tomorrow’s.”15  This 
tradition  continued  with  ATEC  adapting  its  business  practices  to  better  serve  the  Nation 
after  the  11  September  2001  attacks. 

To  respond  to  the  urgent  and  rapid  demand  for  weapons  to  support  the 
deployment  in  Iraq  and  Afghanistan,  ATEC  institutionalized  a  new  reporting  process  that 
stressed  the  identification  and  documentation  of  the  equipment  capabilities  and 
limitations  for  the  decision  makers  and  the  users  prior  to  fielding  to  theater.  Since 
November  2003,  ATEC  has  been  deploying  the  Forward  Operational  Assessment 
(FOA)  team,  on  six-month  rotations,  to  Iraq,  Kuwait,  and  Afghanistan  “to  collect 
information  on  the  systems,  to  identify  shortfalls  and  fix  issues  dealing  with  everything 
from  filling  gaps  in  communication  to  conveying  what  Soldiers  are  actually  requesting  in 
new  equipment  and  capabilities.”16  However,  the  most  profound  change  in  ATEC’s 
culture  has  been  in  a  significant  campaign  to  embrace  and  implement  guidance 
published  in  a  December  2007  memorandum  and  later  incorporated  in  the  revised 
Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense 
Acquisition  System,  8  December  2008. 17 
New  T&E  Policies 

This  memorandum  stated,  “The  fundamental  purpose  of  T&E  is  to  provide 
knowledge  to  assist  in  managing  the  risks  involved  in  developing,  producing,  operating, 
and  sustaining  systems  and  capabilities.  The  goal  is  early  identification  of  technical, 
operational,  and  system  deficiencies,  so  that  appropriate  and  timely  corrective  actions 
can  be  developed  prior  to  fielding  the  system.”18  To  meet  this  goal  the  Director  and  the 
Under  Secretary  decided  to  immediately  implement  several  policies  listed  in  the 
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December  2007  memorandum  awaiting  the  publication  of  the  revised  5000.02 
instructions.  However,  the  philosophy  espoused  in  these  policies  is  not  totally  new.  As 
far  back  as  1986,  the  Blue  Ribbon  Commission  on  Defense  Management  established 
by  President  Reagan  made  remarks  that  advanced  some  of  these  thoughts.  A 
summary  of  these  policies  is  as  follows: 

•  Developmental  and  operational  test  activities  shall  be  integrated  and 
seamless  throughout  the  system  life  cycle 

•  Evaluations  shall  include  a  comparison  with  current  mission  capabilities  using 
existing  data,  so  that  measurable  improvements  can  be  determined. 

•  T&E  should  assess  improvements  to  mission  capability  and  operational 
support  based  on  user  needs  and  should  be  reported  in  terms  of  operational 
significance  to  the  user. 

•  To  maximize  the  efficiency  of  the  T&E  process  and  more  effectively  integrate 
developmental  and  operational  T&E,  evaluations  shall  take  into  account  all 
available  and  relevant  data  and  information  from  contractor  and  government 
sources. 

•  Operational  evaluators  will  continue  to  fulfill  their  statutory  roles  in  providing 
assessments  of  operational  effectiveness,  operational  suitability,  and 
survivability  to  the  Milestone  Decision  Authority.  This  paper  does  not  address 
ATEC’s  implementation  of  this  principle  because  it  falls  in  line  with  the 
mandated  mission  of  the  Army  Evaluation  Center  (AEC)  which  is  one  of  the 
Subordinate  Command  Activities  (SCA). 
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•  To  realize  the  benefits  of  modeling  and  simulation  (M&S),  T&E  will  be 
conducted  in  a  continuum  of  live,  virtual,  and  constructive  system  and 
operational  environments.19 

ATEC’s  Interpretation  of  New  Policies  and  Posture 

Aware  of  the  acquisition  challenges  which  include  procurement  timeline, 
development  of  software  intensive  systems,  and  lack  of  adequate  time  to  conduct 
comprehensive  T&E,  the  ATEC  leadership  fully  embraced  the  OSD  policies.  They 
espoused  aspects  of  a  philosophy  desired  in  Army  T&E  since  1999  with  the 
establishment  of  the  ATEC  organization  which  integrated  DT  and  OT  and  championed 
the  methodology  of  continuous  evaluation.  Early  in  January  2008,  the  leadership 
started  an  aggressive  campaign  to  implement  the  policies  and  published  the  following 
guidelines  to  be  used  by  the  ASTs  for  all  traditional  T&E  efforts  as  applicable: 

•  Include  a  comparison  with  current  mission  capabilities; 

•  Assess  improvements  to  mission  capability  and  operational  support  based  on 
user  needs 

•  Report  in  terms  of  operational  significance  to  the  user; 

•  Conduct  T&E  in  the  mission  context  expected  at  time  of  fielding 

•  Take  into  account  all  available  and  relevant  data  and  information 

•  Continue  to  assess  operational  effectiveness,  operational  suitability,  and 
survivability 

•  Identify  strengths  and  weaknesses  in  meeting  Warfighters’  needs 

•  Realize  the  benefits  of  M&S.20 
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As  an  organization,  ATEC  had  already  underscored  the  principles  expressed  in 
the  new  policies.  Its  mission  statement,  which  aligns  very  well  with  the  policies,  is 
“ATEC  plans,  conducts,  and  integrates  developmental  testing,  independent  operational 
testing,  independent  evaluations,  assessments,  and  experiments  in  order  to  provide 
essential  information  to  decision  makers.”21  Both  the  OSD  memorandum  and  the  DoDI 
5000.02  echo  that  “Evaluations  shall  take  into  account  all  available  and  relevant  data 
and  information  from  contractor  and  government  sources.”22  The  command’s  vision 
highlights  “An  ATEC  that  is  the  premier  test  and  evaluation  organization  within  DoD- 
valued  by  customers  and  decision  makers  for  providing  essential  information  that 
ensures  warfighters  have  the  right  capabilities  for  success  across  the  entire  spectrum  of 
operations.”23  ATEC  leadership  takes  the  appropriate  steps  at  the  appropriate  time  to 
place  the  command  in  the  strategic  position  to  fulfill  this  vision  as  explained  earlier 
about  the  support  to  the  rapid  acquisition  initiatives.  Keeping  with  this  tradition,  the 
leadership  espouses  and  enacts  various  business  practices  and  meaningful  strategic 
initiatives,  in  addition  to  the  systematic  application  of  the  mentioned  guidelines,  to 
posture  the  ASTs  in  meeting  the  DoD  5000.02  intent.  These  include:  a  strong  and 
renewed  early  focus  on  suitability  assessment,  application  of  experimental  design  in 
T&E  planning,  coordination  on  all  T&E  documents  and  products  with  the  program 
manager  (PM)  to  avoid  any  surprises  and  mitigate  high  T&E  costs,  and  most  importantly 
the  implementation  of  the  mission-based  T&E  (MBT&E).24 
Scope  of  MBT&E 

MBT&E  is  the  most  significant  initiative  being  implemented  by  ATEC  and 
provides  the  foundation  for  the  implementation  of  the  policies  published  in  the  OSD 
memorandum  and  captured  in  the  DoDI  5000.02.  It  is  the  pillar  that  represents  “ways” 
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in  ATEC’s  strategy  to  implement  the  elements  of  these  policies.  “Mission-based  T&E 
focuses  on  the  identification  and  alignment  system  components  and  functions  with  the 
tactical  missions  and  warfighting  functions/tasks  that  the  system  supports.”25  The 
practice  of  MBT&E  shifts  the  focus  of  the  T&E  effort  from  the  materiel  attributes 
expressed  in  the  capabilities  documents  and  the  Army  Training  and  Doctrine 
Command’s  (TRADOC)  critical  operational  issues  and  criteria  to  the  operational 
capabilities  necessary  for  the  warfighters  to  perform  their  mission.  It  requires 
coordination  and  buy-in  from  all  key  stakeholders.  The  success  of  this  process 
depends  on  the  AST  members’  knowledge  and  understanding  of  the  tactics  and 
procedures  as  well  as  a  general  operational  understanding  of  the  units’  generic  mission. 
What  additional  norms  can  the  command  and  its  workforce  adopt  to  continue  its 
contributions  to  the  acquisition  process  in  light  of  the  following  five  policy  updates? 

T&E  Policies  Scope  and  Applicability  in  ATEC 

Policy  1  -  Developmental  and  Operational  Test  Activities  Shall  be  Integrated  And 
Seamless  Throughout  the  System  Life  Cycle.  This  policy  aids  in  the  management  of 
the  acquisition  programs  to  control  cost  and  schedule.  It  requires  a  break  from  the 
pervasive  parochial  cultures  entrenched  in  the  acquisition  and  T&E  communities.  The 
Defense  Science  Board  Task  Force  identified,  “There  has  been  reluctance  to  involve 
the  test  and  evaluation  community  early  by  some  program  offices  hoping  to  maintain 
control  of  early  test  results.  There  has  also  been  reluctance  in  some  testing 
organizations  to  be  involved  early  out  of  fear  of  losing  their  independence.”26 
Leadership  throughout  ATEC  and  at  every  level  in  the  SCA  must  challenge  AST  to 
develop  comprehensive  T&E  strategy  that  is  coordinated  and  vetted  with  the  T&E 
Working-level  Integrated  Product  Team  (WIPT)  to  fully  exercise  this  policy. 
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Some  of  the  Warfighting  Functions  in  ATEC  embraced  this  concept  already  and 
used  this  method  in  the  past  due  to  the  system  complexity  as  well  as  program  costs  and 
schedule  challenges.  For  instance,  the  Air  and  Missile  Defense  (AMD)  community  has 
supported  the  implementation  of  T&E  strategy  that  fully  integrated  developmental  and 
operational  tests  since  the  mid  1990’s  for  the  PATRIOT  missile  system.  ATEC  had  also 
developed  similar  strategy  for  the  Theater  High  Altitude  Area  Defense  (THAAD)  and 
National  Missile  Defense  systems.  Early  on  OPTEC/ATEC  and  program  office 
personnel  as  well  as  DOT&E  representatives  realized  the  scope  and  associated  T&E 
costs  for  the  AMD  systems  will  be  broad  and  expensive.  The  stakeholders  embraced 
ATEC’s  strategy  and  agreed  to  coordinate  test  objectives  and  decided  jointly  on  a  test 
configuration  that  meets  each  agency  requirements  through  consensus. 

Consensus  building  is  required  for  ATEC  to  formalize  the  practice  throughout  the 
command  for  every  traditional  acquisition  program.  Mr.  Christopher  DiPetto,  Acting 
Director,  Developmental  Test  and  Evaluation,  Office  of  the  Secretary  of  Defense,  wrote 
in  September  2009,  “I  believe  the  real  obstacles  to  fully  implementing  integrated  testing 
are  mostly  cultural  and  can  be  overcome  with  appropriate  action  by  acquisition 
leaders.”27  As  witnessed  with  the  AMD  programs  it  will  require  leadership,  vision,  and 
creativity  from  the  AST  Chairs  as  well  as  support  through  strategic  communication  from 
the  Directorates,  SCA  leadership  through  the  ATEC  Command  Group. 

Each  stakeholder  may  seek  to  apply  this  policy  for  various  reasons.  It  is 
plausible  that  the  T&E  (WIPT)  may  interpret  this  policy  as  a  reason  for  abbreviated  OT 
events  due  to  resource  scarcity,  potential  for  extended  procurement  time,  and 
challenges  to  get  available  test  units.28  Real  world  obligations  limit  the  availability  of 
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Army  units  to  participate  in  test  events.  This  fact  is  illustrated  in  the  Army  Forces 
Command  having  difficulty  supplying  units  to  support  six  operational  tests  scheduled  to 
take  place  in  2010. 29  However,  the  AST  has  the  responsibility  to  explain  to  the  T&E 
WIPT  that  the  essence  of  this  policy  is  not  to  substitute  developmental  test  for 
operational  test.  Rather,  the  key  idea  is  that  this  policy  allows  for  a  strategy  that 
supports  the  continuous  evaluation  of  the  system  by  generating  data  to:  1 )  facilitate  and 
assess  technology  maturity,  2)  feed  a  reliability  growth  curve  program  for 
comprehensive  suitability  assessment,  and  3)  rehearse  for  Initial  Operational  Test  and 
Evaluation  (IOT&E).30  It  will  take  trust,  will,  and  great  negotiation  skills  from  all  the 
stakeholders  to  build  consensus  to  implement  this  policy.  Simply  put,  it  is 
recommended  that  this  policy  be  implemented  in  a  manner  that  allows  compliance  with 
the  intent  of  Title  10  United  States  Code  (USC)  Section  139  which  dictates  weapons 
must  be  tested  thoroughly  and  realistically. 

Policy  2  -  Evaluations  Shall  Include  a  Comparison  with  Current  Mission 
Capabilities  Using  Existing  Data,  So  That  Measurable  Improvements  Can  Be 
Determined.  This  policy  is  about  comparing  evaluations,  not  the  test  events,  to  discern 
measurable  improvements.  It  does  not  advocate  the  need  for  additional  test  events  if 
the  relevant  data  is  available.  As  a  matter  of  fact,  it  embraces  the  use  of  existing  data 
and  requires  the  AST  to  look  for  opportunities  outside  the  traditional  sources  to  support 
the  evaluation  activities.  It  also  challenges  the  AST  members  understanding  of  the 
military  unit’s  mission  scope  to  facilitate  the  development  of  meaningful  evaluation 
objectives.  The  continuous  evaluation  method  professed  by  AEC  and  the  practice  of 
MBT&E  provide  the  infrastructure  to  fully  implement  this  policy. 
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It  is  the  AST’s  obligation  to  use  creative  and  strategic  thinking  to  identify  venues 
that  facilitate  comparison  of  mission  capabilities.  As  practiced  in  some  instances  in  the 
AMD  community,  the  T&E  WIPT  leverages  systems’  test  events  as  long  as  the 
objectives  could  be  integrated  on  a  non-interfering  basis.  Other  venues,  although 
extremely  difficult  to  leverage,  can  be  training  or  military  exercises.  Using  systems  that 
are  in  development  but  mature,  ready  to  enter  IOT&E  and  for  which  TRADOC  has 
developed  the  tactics,  techniques,  and  procedures  (TTPs),  the  ARFORGEN  training 
cycle  should  provide  an  excellent  realistic  venue  to  observe  and  collect  data  from  the 
units  in  their  operational  configurations.  In  describing  the  ARFORGEN  as  the  Drive 
Train,  “The  Army  focuses  units  against  future  missions  early  and  task  organizes 
modular  expeditionary  forces  tailored  to  mission  requirements.  ...operational 
requirements  drive  the  ARFORGEN  training  and  readiness  process.”31  This  leverage  of 
the  ARFORGEN  can  also  alleviate  the  challenge  in  the  lack  of  military  unit’s  availability 
due  to  real  world  obligations  to  execute  the  tests.  However,  particular  coordination  and 
planning  must  take  place  prior  to  allow  ATEC’s  participation  in  these  activities.32 
Approval  will  require  Army’s  senior  leaders’  involvement  to  include  ATEC’s  and 
participating  Operating  Force’s. 

In  the  meantime,  the  deployment  of  the  Army  units  to  Iraq  and  Afghanistan 
provide  the  best  opportunity  to  evaluate  mission  capabilities  in  the  most  realistic 
environment  for  systems  deployed  in  these  theaters.  ATEC  has  been  in  the  forefront  to 
support  and  to  take  advantage  of  this  operational  environment  through  the  FOA  team. 
This  new  paradigm  differs  greatly  from  the  traditional  operational  testing  performed  at 
the  test  ranges  to  support  the  acquisition  efforts.  The  team  is  embedded  with  various 
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selected  units  in  theater  to  observe  and  collect  real-time  information,  in  a  non-intrusive 
fashion,  as  the  conflict  takes  place.  “Through  much  coordination  with  U.S.  forces, 
coalition  partners,  and  multiple  civilian  organizations,  ATEC's  FOA  teams  have  provided 
the  eyes,  ears,  and  knowledge  to  Army  senior  leaders  with  often  the  only  current  and 
relevant  information  with  which  to  make  decisions.”33  Ultimately  the  AEC  personnel 
can  use  this  data  to  perform  the  evaluation  and  document  the  systems’  existing 
capabilities  demonstrated  in  an  operational  environment.  This  evaluation  will  constitute 
the  mission  capabilities  baseline  against  which  future  system  improvements  or  systems 
with  similar  capabilities  procured  for  similar  units  can  be  compared.  Having  established 
this  baseline,  the  AST  must  remain  aware  that  the  tactics  and  procedures  used  in  Iraq 
and  Afghanistan  are  tailored  for  this  specific  environment  to  fight  the  specific  enemy. 
Therefore,  as  the  user  adjusts  the  tactics  and  the  concept  of  operations,  the  AST  must 
adapt  their  strategy  and  take  these  changes  into  consideration  to  properly  conduct  the 
comparable  evaluation. 

Policy  3  -T&E  Should  Assess  Improvements  to  Mission  Capability  and 
Operational  Support  Based  On  User  Needs  and  Should  Be  Reported  in  Terms  of 
Operational  Significance  to  the  User  -  Evaluation  Conducted  in  the  Mission  Context. 

The  first  observation  is  this  policy  statement  focuses  on  “mission  capabilities.”  This 
focus  represents  a  key  shift  from  previous  policies  where  the  T&E  emphasis  was  on 
verifying  if  the  equipment  could  perform  the  user  requirements  to  include  the  key 
performance  parameters.  This  policy  statement  is  long  overdue  and,  to  quote  Mr.  Steve 
Daly,  “Mission  success  is  not  about  the  specifications  -  not  about  the  “shalls.”  The 
units,  not  the  equipment,  perform  the  mission.”34  This  entails  evaluation,  as  the 
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equipment  mature  and  the  various  support  packages  become  available,  that  is  to  be 
based  on  data  generated  from  events  where  the  equipment  is  manned  with  trained 
military  personnel,  to  include  operators  and  maintainers,  performing  a  representative 
mission  in  accordance  with  approved  TTPs.  Second,  this  policy  also  mandates 
assessment  of  “operational  support  based  on  user  needs.”  This  requires  an 
assessment  of  operational  suitability  to  include  the  whole  logistics  trail,  wartime  usage 
rates,  reliability,  and  maintainability.  The  community  has  to  question  how  realistic  is  this 
policy;  and  has  to  be  willing  to  exercise  common  sense  to  ensure  the  associated  cost 
and  necessary  schedule  allotment  do  not  outweigh  the  benefits.  There  are  some 
significant  challenges  and  limitations  for  any  Service  OTA  to  fully  implement  this  policy. 

The  intentions  expressed  in  this  policy  are  fully  met  in  ATEC’s  MBT&E  which 
enables  evaluation  in  the  mission  context.  One  key  advantage  offered  by  the  MBT&E 
methodology  is  the  use  of  the  Army  Universal  Task  List  which  is  a  comprehensive,  but 
not  all-inclusive,  listing  of  Army  tasks,  missions,  and  operations  for  the  various 
warfighting  functions.35  The  use  of  the  task  list  facilitates  a  doctrinal  reference  and 
helps  establish  measures  and  standards  used  in  the  evaluation.36  In  addition,  the  AST 
should  work  very  closely  with  the  Directorate,  Combat  Development  representatives  to 
clarify  and  set  boundaries  for  a  typical  mission.  Cooperation  and  consensus  must  be 
reached  among  the  participants  to  identify  the  appropriate  objectives,  based  on 
software  and  materiel  maturity,  for  each  phase  of  testing  and  evaluation. 

Another  challenge  to  fully  implement  this  policy  is  that  the  US  military  usually 
prosecutes  wars  as  a  joint  team  or  even  part  of  a  multinational  force.37  It  is  unrealistic  to 
develop  a  T&E  strategy  to  capture  comprehensive  missions  to  be  performed  by  a  single 
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military  unit.  Test  events  are  experiments  that  can  never  replicate  the  fog  and  friction  of 
war  as  postulated  by  Clausewitz.38  T&E  professionals  need  to  accept  this  limitation. 

The  leadership  throughout  the  chain  of  command  must  recognize  these  challenges  and 
limitations,  and  be  willing  to  live  with  them.  Realistic  expectations  must  be  established 
by  the  AST  and  the  T&E  WIPT.  The  ATEC  leadership  will  continue  to  communicate 
these  expectations  to  the  Army  and  OSD  Staff. 

Policy  4  —  To  Maximize  the  Efficiency  of  the  T&E  Process  and  More  Effectively 
Integrate  Developmental  And  Operational  T&E,  Evaluations  Shall  Take  Into  Account  All 
Available  and  Relevant  Data  and  Information  From  Contractor  and  Government 
Sources.  The  approval  to  use  available  and  relevant  contractor  data  has  never  been 
stated  with  such  clarity  in  any  policies  from  DOT&E  or  as  stated  in  this  version  of  the 
DoDI  5000.02.  This  policy  represents  the  biggest  departure  from  previous  T&E  policies. 
This  statement  appeared  in  two  different  locations  in  the  instruction.  The  Defense 
Science  Board  reported  in  1999  that  “The  vast  majority  of  test  objectives  (80%)  provide 
developmental  insights  as  well  as  operationally  relevant  information.”39  The  operating 
word  in  the  policy  is  relevant.  There  usually  are  plenty  of  data  from  all  the  sources; 
however,  the  questions  have  to  be  are  they  reliable,  were  they  generated  in  the  right 
context,  and  were  they  properly  collected  and  reduced?  This  policy  provides 
tremendous  flexibility  for  the  Service  OTA  to  work  early  with  the  PMs  to  shape  the 
contractor  tests  by  documenting  their  needs  and  requests  for  the  contractor  data.  The 
AST  early  involvement  required  by  ATEC  leadership  is  essential  in  implementing  this 
policy. 
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A  catalyst  in  this  process  is  a  Test  and  Evaluation  Master  Plan  (TEMP) 
coordinated,  vetted,  and  approved  by  a  T&E  WIPT  to  document  all  stakeholders’ 
requirements.40  However,  in  a  2  September  2008  memorandum,  Department  of  Army 
representatives  wrote  “Our  30-day  standard  for  a  HQDA  approved  TEMP  is  still  not 
always  being  achieved.”41  They  recommend  empowerment  of  the  T&E  WIPT  members, 
involvement  of  management  from  all  levels,  open  and  honest  working  relationships,  and 
early  initiation  of  the  formal  approval  process  to  improve  the  TEMP  approval  timeline.42 
As  the  proponent  of  the  TEMP,  the  Materiel  Developer  is  responsible  to  ensure  this 
overarching  strategic  plan  is  approved  early  and  should  be  incentivized  to  use  military 
subject  matter  experts,  to  include  operators  and  maintainers,  during  their  DT  to  provide 
feedback  regarding  the  design’s  potential  military  usefulness. 

Additionally,  the  current  instruction  states  "The  lead  OTA  shall  brief  the  DOT&E 
on  concepts  for  an  OT&E  120  days  prior  to  start.  They  shall  submit  the  OT&E  plan  60 
days  prior,  and  shall  report  major  revisions  as  they  occur."43  To  successfully  create  an 
environment  to  generate  data  that  are  relevant  to  differing  stakeholders  there  must  be  a 
prerequisite  step  much  earlier  than  these  timelines.  Coupled  with  ensuring  the  TEMP  is 
forwarded  for  approval  in  a  timely  manner,  a  path-ahead  should  be  to  require  the  T&E 
WIPT,  not  just  the  Lead  OTA,  to  brief  key  stakeholders,  to  include  OSD  for  oversight 
programs,  180  days  after  a  system  becomes  a  program  of  record  on  their  T&E  strategy. 
This  briefing  should  focus  mainly  on  the  test  events  to  be  conducted  through  IOT&E  to 
include  contractor  tests,  government  DT,  and  OT.  This  will  provide  an  opportunity  to  get 
early  feedback  and  buy-in  from  the  senior  stakeholders.  Although  each  T&E  WIPT 
member  has  the  responsibility  to  inform  his  leadership  of  the  issues,  this  forum  will  allow 
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a  collective  or  holistic  presentation  of  the  test  events  and  how  each  stakeholder  will  be 
getting  the  data  needed  to  comply  with  this  principle.  This  approach  will  force  the  T&E 
WIPT  members  to  get  involved  early  in  identifying  their  requirements  and  therefore 
bring  all  the  contemptuous  issues  and  challenges  up  early  for  resolution. 

Policy  5  -  To  Realize  the  Benefits  of  Modeling  and  Simulation  (M&S),  T&E  Will 
Be  Conducted  In  a  Continuum  of  Live,  Virtual,  and  Constructive  System  and 
Operational  Environments.  It  is  well  documented  that  M&S  provides  a  series  of 
advantages.  However,  one  must  appreciate  the  fact  that  M&S  are  just  a  representation 
of  capabilities  or  functionalities.  They  should  not  be  used  just  for  convenience.  The 
Defense  Science  Task  Board  highlighted,  “Substantial  early  investment  is  often  needed 
to  capitalize  on  the  potential  M&S  benefits  in  the  T&E  process.”44  The  development  and 
use  of  these  tools  can  be  time  consuming  and  expensive.  It  requires  expertise  and 
knowledge  of  the  system  as  well  as  the  TTPs  to  build  an  accurate  representation.  The 
fidelity  of  an  M&S  tool  impacts  the  cost  associated  with  the  development  of  that  tool. 

The  verification,  validation,  and  accreditation  of  a  model  rely  on  the  quality  and  the 
reliability  of  the  data  used  to  perform  these  functions.  Describing  the  need  to 
understand  how  M&S  contribute  to  T&E  the  Task  Force  reported,  “Require  the  users  of 
M&S  in  their  evaluations  to  state  model  assumptions,  limitations,  and  uncertainties 
explicitly,  as  well  as  sources  for  input  data  in  the  presentations  of  results  to  decision 
makers.”45  Therefore,  here  are  three  recommended  critical  takeaways  the  AST  must 
follow  when  implementing  this  policy:  1)  determine  if  the  benefits  outweigh  the  costs;  2) 
ensure  the  accreditation  of  the  tool  is  focused  on  the  fidelity  of  the  functionalities 
modeled  to  facilitate  an  accurate  assessment  regarding  the  mission  context,  operational 
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realism,  as  well  as  relevance;  3)  understand  the  what,  why,  where,  and  when 
associated  with  the  system  requirements  to  be  tested  and  evaluated  using  the  tool. 

The  ATEC  community  uses  M&S  when  appropriate  routinely.  M&S  events  and 
hardware  in  the  loop  tests  (HWILT)  are  necessary  phases  in  every  ATEC  T&E  strategy 
for  the  AMD  systems.  The  reality  and  the  complexity  associated  with  these  systems  do 
not  allow  the  AST  to  escape  this  axiom.  ATEC  conducted  the  first  HWILT  with  a  joint 
architecture  at  the  Joint  National  Test  Facility  in  2002  to  test  interoperability  during  the 
Patriot  Advanced  Capability  3  IOT&E.  The  event  was  sponsored  by  the  Missile 
Defense  Agency  and  included  capabilities  that  represented  key  major  theater  missile 
defense  systems  to  include  early  warning  sensors  and  command  and  control  nodes. 

The  Patriot  AST  also  uses  the  flight  mission  simulator,  among  other  modeling  and 
simulation  tools,  routinely  to  support  their  continuous  evaluation  activities.46  The  plan  is 
for  Raytheon  Corporation  to  develop  a  HWIL  facility  conceived  after  the  one  used  in  the 
Patriot  program  for  the  Surface  Launched  Advanced  Medium  Range  Air-to-Air  Missile 
IOT&E.47  The  AST  for  the  Army  Integrated  Air  and  Missile  Defense,  a  pre-major 
defense  acquisition  program  effort,  is  working  with  the  PM  on  a  T&E  strategy  that  will 
take  full  advantage  of  their  M&S  tools  and  facilities.48 
ATEC  Current  State  for  New  Policies 

Some  sectors  in  ATEC  have  been  guided  by  some  of  the  principles  espoused  in 
these  new  policies  before  OSD  published  them.  Dr.  James  Streilein  and  Ms  Luna 
pointed  out  that  “the  Ballistic  Missile  Defense  System  T&E  strategy  features  some  of 
these  policies  in  ATEC  and  they  also  highlighted  the  M915A5,  a  commercial  purchase 
of  a  line  haul  semi-tractor,  and  the  Mine  Resistant  Ambush  Protected  vehicle  programs 
as  other  examples  where  AST  is  using  integrated  DT  and  OT.”49  Where  is  ATEC  in  fully 
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implementing  these  principles  throughout  the  Command?  Upon  publication  of  the  OSD 
memorandum,  ATEC  conducted  an  initial  assessment  which  revealed  that  one  of  the 
five  principles  examined  in  this  paper,  measuring  improvement  to  mission  capability, 
had  been  fully  embedded  in  ATEC’s  processes;  and  the  other  principles  were  either 
partially  embedded  or  not  embedded  in  the  ATEC’s  processes.50  The  leadership  has 
since  reported  twice  on  the  state  of  implementation  of  these  principles  in  the 
Command’s  business  practices.  It  needs  to  be  noted  that  the  second  policy  was  broken 
into  two  categories  to  facilitate  tracking  improvements  to  capability  and  operational 
support  as  well  as  tracking  comparisons  of  separate  current  mission  capabilities.  The 
first  review  reflected  activities  captured  for  nine  systems.  It  showed  that  one  policy  ~ 
use  of  all  available  information  ~  was  fully  addressed  in  all  nine  T&E  efforts.  The  other 
policies  were  either  addressed  partially  or  not  at  all.  The  second  review  for  twelve 
different  systems  revealed  that  measuring  improvement  to  capability  and  operational 
support  and  integration  of  DT  and  OT  still  lags  the  implementation  of  the  other 
policies.51  How  can  ATEC  bridge  this  gap? 

Path  Ahead 

Although  the  nature  of  the  working  relationship  in  the  T&E  IPT  can  be  one  of 
collaboration,  it  is  plausible  to  argue  there  is  need  for  improvement  in  removing 
parochialism  and  cultural  barriers.  The  ATEC  leadership  has  been  fostering  a  more 
cooperative  environment.  The  AEC  Director  mandated  that  the  AST  invite  the  PM  to 
attend  and  participate  in  T&E  strategy  presented  to  ATEC.  It  is  CG  ATEC’s  policy  that 
the  AST  coordinates  all  T&E  documents  and  products  with  the  PM  to  avoid  any 
surprises  and  mitigate  high  T&E  costs. 
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A  common  obstacle  to  effective  management  and  building  consensus  in  many 
organizations  is  the  reluctance  of  personnel  to  elevate  issues  to  the  supervisors  and 
senior  leaders  for  resolution.  This  can  result  in  wasting  valuable  time  while  attempting 
to  solve  problems  that:  1 )  are  beyond  the  control  of  action  officers,  and  2)  within  senior 
leaders’  ability  to  resolve  easily  based  on  relationship  or  having  a  broader  view  of  the 
multitude  of  challenges  the  organizations  and  the  Army  are  facing.  ATEC  regulation 
prescribes  established  timelines  and  milestones  for  various  reviews  to  be  conducted 
throughout  the  T&E  program.  These  milestones  are  also  listed  in  the  Section  IV  of  each 
event  Test  Resource  Plan.52  It  is  up  to  the  staff  and  Directorate  leadership  to  take 
advantage  of  these  forums  and  ensure  that  issues  preventing  or  impacting  a  full 
implementation  of  these  policies  are  identified  and  presented  with  appropriate  courses 
of  action  to  the  Senior  Leaders  during  these  reviews.  Decision  to  approve  a  course  of 
action  by  the  leadership  empowers  the  AST  to  negotiate  with  the  T&E  WIPT.  Knowing 
the  path  the  ATEC  leadership  supports  places  the  AST  in  a  better  position  to  engage  in 
consensus  building  with  their  counterparts. 

A  consistent  message  from  speakers  to  the  Army  War  College  Academic  Year 
2010  class  is  to  think  “Army  Enterprise  and  Army  goals”.53  The  Army  Enterprise 
“encourages  Civilian  and  Military  leaders  to  take  a  holistic  view  of  Army  objectives  and 
resources,  and  empowers  them  to  integrate  related  functions  to  effectively  and 
efficiently  generate  trained  and  ready  forces  for  Combatant  Commanders  and  sustain 
the  All-Volunteer  Force.”54  The  path  ahead  for  ATEC  leadership  throughout  the 
Command  is  to  continue  to  train  and  demand  that  the  AST  members  think  strategically. 
Parochialism  can  be  a  handicap,  not  a  facilitator.  The  message  needs  to  be  one  team 
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and  one  mission,  the  Army  team.  The  infusion  of  new  employees  in  the  AEC  due  to 
retirement  and  the  ongoing  geographical  relocation  presents  an  excellent  opportunity  to 
plant  this  seed  in  the  workforce.  However,  it  will  take  more  than  willingness  from  ATEC 
to  lead  and  petition  for  the  need  of  consensus. 

The  various  organizations  represented  in  the  T&E  WIPT  must  be  willing  to  join 
ATEC  in  reaching  consensus  toward  the  best  possible  means  to  implement  the  five  T&E 
policies  discussed.  The  purpose  is  to  target  not  only  unity  of  command  but  also  unity  of 
effort  to  become  more  efficient  in  delivering  the  most  effective,  suitable,  and  survivable 
capabilities  to  the  units  to  conduct  their  mission.  The  Materiel  Developer  community 
has  to  face  the  reality  that  many  of  the  systems  being  developed  are  complex  and 
software  intensive.  These  systems  require  adequate  and  possibly  extensive  testing  to 
identify  and  mitigate  the  inherent  risks  associated  with  pursuing  these  developments. 
The  community  has  to  embrace  the  concept  that  “test  “failures”,  especially  in  the  early 
phases  of  system  development,  should  be  received  as  important  learning  opportunities 
and  chances  to  solve  problems  as  they  are  uncovered.”55  Therefore,  a  path  ahead 
should  be  for  the  Combat  Developer  to:  1 )  establish  realistic  requirements  and  2)  be 
willing  to  accept  less  than  the  100  percent  solution  at  the  first  fielding.  Dr.  James  J. 
Streilein,  Technical  Director  for  ATEC,  concluded  “success  in  fielding  equipment  to  our 
warfighters  will  continue  to  require  total  commitment,  coordination,  and  cooperation  of 
all  members  of  the  acquisition  communities.  I  have  seen  the  T&E  community 
continually  improve  over  the  years  since  1974,  and  I  look  forward  to  our  efforts  and 
innovations  to  handle  the  increases  in  complexity  of  systems  to  be  tested  and  evaluated 


21 


in  the  future.”56  This  is  the  mindset  that  is  influencing  positively  the  posture  of  the  ATEC 
organization. 

Conclusion 

As  OSD  establishes  new  acquisition  and  T&E  policies,  the  main  purpose  of  T&E 
remains  to  provide  unbiased  and  relevant  information  to  the  decision  makers  to  manage 
risks.  Anticipating  the  need  for  integrated  T&E,  the  Army  approved  the  consolidation  of 
DT  and  OT  in  1998.  The  ATEC  leadership  has  consistently  shown  flexibility  and 
adaptability  in  approving  T&E  strategy  that  calls  for  use  of  M&S  as  well  as  all  available, 
relevant,  and  credible  data  to  support  the  evaluation  of  the  missile  defense  systems. 

The  ATEC  leadership  has  embraced  the  principles  espoused  in  the  T&E  policies 
published  by  OSD  and  has  implemented  various  initiatives  to  facilitate  their  full 
implementation.  ATEC’s  key  strategic  initiative  is  the  MBT&E.  Success  in  fulfilling 
these  policies  is  a  factor  of  trust  and  the  community’s  will  to  build  consensus. 

Consensus  among  all  stakeholders  in  accepting  the  limitations  associated  with  the 
development  and  testing  of  complex  systems  are  necessities  of  the  time.  The  T&E 
WIPT  members  must  be  bold  and  show  the  desire  to  break  their  agencies’  cultural 
barriers.  These  members  must  challenge  their  leadership  and  adopt  norms  such  as 
critical  and  creative  thinking  and  collaboration  for  unity  of  effort.  The  first  critical  step  is 
for  all  stakeholders  to  embrace  the  thought  that  although  their  agency  has  specific 
mission  and  functions  that  contribute  in  the  overall  acquisition  process,  the  objective  is 
not  about  them.  The  agencies  are  facilitators  and  contributors;  the  objective  is  to  equip 
the  Warfighters. 
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